Jan 12
5
Posted by randfish If you've been following my posts on Linkscape's index, you know that we've been trying to aim for fresher, better and larger indices over the past few months, but have been finding some very tough challenges. It turns out that indexing the web, canonicalizing millions of pages and calculating a link graph with quality metrics is super-hard; who knew? 
As part of those efforts, we've been working toward an experimental index that leverages a more search-engine style crawler that crawls fresher pages/sites more often and less fresh stuff less frequently.
Read the original here:
Interim Linkscape Update for January